Data is being captured, stored, tagged, indexed, mined, and consumed at alarming rates. Moreover, advancements in network connectivity and network bandwidth have permitted data to be omnipresent in our daily lives.
Data streaming permits individuals or enterprises to subscribe to data feeds (e.g., news feeds, business feeds, sports feeds, entertainment feeds, political feeds, etc.) and receive information on whatever device individuals prefer, such as phones, laptops, computers, wearable processing devices, and the like.
One problem with data streaming is that raw data from data feeds often progresses from original source feeds through a variety of intermediate processing sites before the final versions of the original data feeds reach the desired consumers. These intermediate processing sites can perform many value-added adjustments to the original source data, such as: filtering out some data that the consumer is not interested in, reformatting some data so it can be easily viewed and digested by the consumer on the consumer's desired device, aggregating some data with other data (such as metrics about the data, overlays of the data with other data, integrating data from a different data feed, and the like).
The intermediate processing sites can become bottlenecks in the timely delivery of information to the consumer especially when the processing sites enhance multiple types of data feeds. Moreover, even when a particular intermediate site designed to perform a particular data feed enhancement is replicated over the network for processing throughput efficiency, there is little to no coordination between different types of intermediate sites to ensure overall processing efficiency associated with delivering of a data feed from a source feed through multiple different types of intermediate processing sites to the consumer. This is so because often different entities are associated with different intermediate processing sites. So, even if one entity controlling one type of intermediate processing site effectively achieves processing throughput, there is still no guarantee that the consumer will receive the information in any more of a timely fashion. In fact, the consumer may actually experience further delay in receiving timely information if a next intermediate site from the processing efficient site becomes overloaded with processing because the processing efficient site is delivering data too rapidly for the next intermediate site to manage and process.
Furthermore, even assuming a source data feed and all its intermediate processing sites are capable of effectively coordinating processing throughput efficiencies, the efficiency may not be what is expected by the various entities because there is very little useful data analytics being presently gathered and shared by these entities. So, the assumption that merely adding more hardware and network connectivity can dramatically improve processing throughput efficiency is likely an incorrect assumption.
Thus, before intelligent decisions can be deployed to improve data feed delivery through a network of intermediate processing sites to a consumer, there needs to be better mechanisms for gathering real-time data analytics and adjusting for changes in the analytics in a dynamic and real-time fashion. This is so because network conditions dynamically change in real time, such that any static based processing improvement decision may be only a temporary patch before conditions change and the improvement becomes a less viable option than what existed before the improvement.